Speech Input in Multimodal Environments: A Proposal to Study the Effects of Reference Visibility, Reference Number, and Task Integration

A model of complimentary behavior has been suggested based on arguments that direct manipulation and speech recognition interfaces have complimentary strengths and weaknesses. Specifically, anecdotal arguments have been given that direct manipulation interfaces are best used for specifying simple actions when all references are visible and the number of references is limited, while speech recognition interfaces are better at specifying more complex actions when references are numerous and not visible. Based on these observations, an empirical study has been proposed to evaluate speech recognition as a complimentary input channel in multimodal direct manipulation/speech recognition environments. It is hypothesized that this complimentary behavior is only true under certain conditions related to reference attributes and the interface tasks. The particular focus of this proposal is to study the effects of reference visibility, reference number, and task integration to characterize the complimentary behavior of speech and direct manipulation.